WiFi remote displays

ABSTRACT

A wireless peripheral mode is provided by a host system that communicates to a WiFi infrastructure and, utilizing the same WiFi RF subsystem, also communicates to peripherals. The host system may employ additional RF channels for communicating with high bandwidth peripherals, such as display devices, where high levels of QoS may be managed locally. The host system may be a conventional desktop computer system, a notebook computer system, a multi-media access point, a cell phone, a game machine, a portable game machine, a Personal Digital Assistant (PDA), a smart phone or any other type of device that benefits from accessing both a WiFi infrastructure and local peripherals.

PRIORITY

This application is a continuation of U.S. patent application Ser. No.14/274,490, filed May 9, 2014, entitled “WiFi Remote Displays,” now U.S.Pat. No. 9,344,237, which is a continuation of U.S. patent applicationSer. No. 13/622,836, filed Sep. 19, 2012, entitled “WiFi RemoteDisplays,” now U.S. Pat. No. 8,732,328, which is a continuation of U.S.patent application Ser. No. 13/225,532, filed Sep. 5, 2011, entitled“WiFi Peripheral Mode,” now U.S. Pat. No. 8,296,453, which is acontinuation of U.S. patent application Ser. No. 11/230,872, filed Sep.19, 2005, entitled “WiFi Peripheral Mode Display System,” now U.S. Pat.No. 8,019,883, which (1) claims the benefit of priority to U.S. PatentProvisional Application Ser. No. 60/705,153, filed Aug. 4, 2005,entitled “WiFi Peripheral Mode,” (2) is a continuation-in-part of U.S.patent application Ser. No. 11/139,149, filed May 27, 2005, entitled“Multiple Remote Display System,” and (3) is a continuation-in-part ofU.S. patent application Ser. No. 11/122,457, filed May 5, 2005, entitled“Computer System For Supporting Multiple Remote Displays,” now U.S. Pat.No. 7,667,707.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to a computer system forwirelessly connecting peripheral devices and more particularly tomethods for wirelessly connecting a host computer to supporting WiFiperipherals and for wirelessly supporting additional display systems.

2. Description of the Background Art

In recent years, there has been growing popularity for the WiFi standardfor wireless communication. WiFi is a collection of standards of whichthe main standards are 802.11g (which incorporates 802.11b) and 802.11a.Within the WiFi standard there are numerous required and optional modesas well as a continually advancing set of enhancements and proposedenhancements. WiFi can be found in practically every new portablecomputer and in an increasing number of desktop computers, telephones,televisions, digital media players and cellular phones. WiFi based LocalArea Networks (WLANs) are found in most airports, coffee shops,businesses, college campuses, city centers and an increasing number ofother areas.

Developing effective methods for operating wireless electronicperipherals is a significant challenge for modern system designers. In atypical system, wireless peripherals, such as a keyboard and mouse, usededicated RF chips with their own RF channels to communicate with acomputer. This adds cost to the computer system and if, as is typical,the computer does not integrate the RF chips for peripheral devices,requires the use of an external dongle.

Recently, computers and cell phones have utilized the Bluetooth standardfor supporting a variety of peripherals including keyboards, mice andtelephone headsets. Bluetooth is a more general standard that is morereadily integrated directly into computers and cell phones. However,Bluetooth still adds a dedicated RE subsystem and since Bluetooth usesthe same RF spectrum as WiFi's 802.11g, additional care must be taken toavoid interference between the two. Other power or cost sensitivedevices, such as the Sony Play Station Portable (PSP) may include only aWiFi RF subsystem and the most convenient method for adding wirelessperipherals is to provide wireless peripherals supporting WiFi.

A device such as a computer on a WiFi network has two modes ofoperation, Infrastructure mode and Ad-Hoc mode. Infrastructure modeallows the computer to communicate with the Access Points (AP) of a WiFinetwork and gain access to services such as the Internet. Ad-Hoc modeallows devices to talk directly to each other without utilizing the APs.These modes are mutually exclusive so a device can not talk to both anAP and another device during the same session.

Mesh Networking is a technique for wireless communications where,instead of a conventional hub-and-spoke model with every deviceconnecting to a central access point, every device (area) acts as arepeater or router. When mesh-enabled devices are in range of oneanother they automatically create a wireless mesh network, and traffichops from device to device until it reaches the nearest AP that includeswired backhaul. Mesh networking reduces the need for more powerfulcentral AP antennas and improves wireless coverage. One downside of meshnetworking is that it increases the minimum processing capability neededby each mesh device. While adding mesh capability typically would not bea problem for a computer, adding conventional mesh capability forperipherals and in particular for low cost battery based peripherals isa large burden.

For high bandwidth and time critical communication, such as between acomputer and a computer display, it may be particularly desirable to addRF capacity beyond the RF channel used for WiFi infrastructure. If themore general network traffic is not normally part of the same RIFchannel then such a channel can operate in a high QoS mode. Options forthe additional RE channel may include an additional channel of WiFi(WiFi supports multiple non-overlapping channels), a WiFi channel in adifferent RF band (802.11a operates in the 5 GHz spectrum while 802.11goperates in the 2.4 GHz spectrum) or a different RF protocol such asUltra Wide Band (UWB).

Adding additional RF channels to the host system is best done forperipheral devices that need high bandwidth and high QoS. For lowbandwidth low cost peripherals it is not desirable to either addadditional RF channels or increase the complexity of the WiFi features.However, standard WiFi does not support a host system that canconcurrently operate a single WiFi channel for both infrastructure modeand peripheral support. Therefore, what is needed is an effectiveWireless architecture where a host system can maintain the bestavailable mode of access to the WiFi AP infrastructure whileconcurrently utilizing low cost peripherals which also use the WiFi RFchannel where any additional RE channels added to the system are usedfor high performance connections.

SUMMARY OF THE INVENTION

The present invention provides an efficient architecture for amulti-display system. In one embodiment, initially, a multi-displaycomputer system sharing one host computer system provides one or moreremote display systems with interactive graphics and video capabilities.The host computer system generally manages frames that correspond toeach remote display system and manages updating the remote displaysystems over a network connection. Three preferred embodiments arediscussed in detail, and many variations are discussed in lesser detail.

In a first preferred embodiment, a host computer utilizes somecombination of software and conventional graphics processing blocks tosupport multiple displays. The graphics processor is configured for avery large frame size or some combination of frame sizes that aremanaged to correspond to the remote display systems. The softwareincludes a tracking software layer that can track when the framecontents for each display, the surfaces or subframes that comprise eachframe, and potentially which precincts or blocks of each surface, areupdated. The encoding process for the frames, processed surfaces orsubframes, or precincts of blocks, can be performed by some combinationof the CPU and one of the processing units of the graphics processor.

In a second preferred embodiment, a host computer utilizes a traditionalgraphics processor whose display output paths, normally utilized forlocal display devices, are connected to a multi-display processor.Supporting a combination of local and remote displays is possible. Forremote displays, the graphics processor is configured to output multipleframes over the display output path at the highest frame rate possiblefor the number of frames supported in any one instance. Themulti-display processor, configured to recognize the frameconfigurations for each display, manages the display data at the frame,scan line, group of scan line, precinct, or block level to determinewhich remote displays need which subfrarne updates. The multi-displayprocessor then encodes the appropriate subframes and prepares the datafor transmission to the appropriate remote display system.

A third preferred embodiment integrates a graphics processor and amulti-display processor to optimize a system configuration. Thisintegration allows for enhanced management of the display frames withina shared RAM where the graphics processor has more specific knowledgefor each frame. Additionally, the sharing of RAM allows themulti-display processor

In each system, after the data is encoded, a network processor, or CPUworking in conjunction with a simpler network controller, transmits theencoded data over wired and/or wireless network(s) to a remote displaysystem(s). Each remote display system decodes the data intended for itsdisplay, manages the frame updates and performs the necessary processingfor the display screen. Other features, such as masking packets lost innetwork transmission, are managed by the remote display system(s). Whenthere are no new frame updates, the remote display controller refreshesthe display screen with the data from the prior frame.

The system may feed back network information from the various wired andwireless network connections to the host system CPU, frame managementand data encoding systems. The host system uses the network informationto affect the various processing steps of producing display frameupdates and, based on the network feedback, can vary the frame rate anddata encoding for different remote display systems. Additionally, forsystems that include noisy transmission channels as part of the networksystem, the encoding step may be combined with forward error correctionprotection in order to prepare the transmit data for the characteristicsof the transmission channel. The combination of these steps produces anoptimal system for maintaining an optimal frame rate with low latencyfor each of the remote display systems.

The present invention further provides a method for implementing a“peripheral mode” or “WiFi-P mode” for wireless systems where a hostsystem can maintain the best available mode of access to the WiFi APinfrastructure while concurrently utilizing low cost peripherals whichalso use the WiFi RF channel. Additionally, the host system may employadditional RF channels for communicating with high bandwidthperipherals, such as display devices, where high levels of QoS may bemanaged locally. The host system may be a conventional desktop computersystem, a notebook computer system, a multi-media access point, a cellphone, a (portable) game machine, a Personal Digital Assistant (PDA), asmart phone or any other type of device that benefits from accessingboth a WiFi infrastructure and local peripherals.

Each host system may utilize whichever capabilities of the WiFiinfrastructure mode device that it chooses. For example, the host maysupport advanced operating modes, such as the Point CoordinationFunction (PCF) or extensions to the standard such as 802.11n forMultiple Input and Multiple Output (MIMO) antennas, 802.11e for QoS, orfor extending the reach of the WiFi network with repeater capabilitiesor extensions such as 802.11s for mesh capabilities. While operating inwhat the invention refers to as “WiFi-Peripheral” (WiFi-P) mode, theperipheral devices, without directly utilizing the WiFi APinfrastructure, communicate with the host system. Simpler devices suchas a keyboard or mouse may be designed to use only the WiFi-P mode tocommunicate with a host, and could not communicate with a WiFiinfrastructure device which did not host a WiFi-P mode. Moresophisticated devices such as a wireless display may support standardWiFi modes, WiFi-P modes, modes which include additional RIF channels,or some combination of modes. Other enhancements for low power operationof peripherals are also supported.

Therefore, the present invention effectively implements an enhancedmulti-display system and a flexible wireless peripheral mode thatutilizes various heterogeneous components to facilitate optimal systeminteroperability and functionality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a multi-display computer system including ahost system connected over one or more networks to various peripheralsand network elements;

FIG. 2 is a block diagram of a host computer of a multi-display systemin accordance with one embodiment of the invention;

FIG. 3 shows a remote display corresponding to FIG. 2 in accordance withone embodiment of the invention;

FIG. 4 represents a memory organization and the path through a dualdisplay controller portion of a graphics and display controller inaccordance with one embodiment of the invention;

FIG. 5 represents a memory and display organization for various displayresolutions in accordance with one embodiment of the invention;

FIG. 6 shows a multi-display processor for the head end system of FIG. 2in accordance with one embodiment of the invention;

FIG. 7 is a block diagram of an exemplary graphics controller with anintegrated multi-display support in accordance with one embodiment ofthe invention;

FIG. 8A is a data flow chart for how subband encoded frames of displayproceed thorough the processing steps in accordance with one embodimentof the invention;

FIG. 8B is a block diagram of graphics and display system enhanced bycombining a graphics and display controller with a display encoder;

FIG. 9A is a flowchart of steps in a method for performing set up,arbitration and transfers with one embodiment of the invention;

FIG. 9B represents the elements and communication channels correspondingto FIG. 9A with one embodiment of the invention;

FIG. 9C is a flowchart of steps in a method for performing multi-displaywindowing, selective encoding and selective transmission, in accordancewith one embodiment of the invention; and

FIG. 10 is a flowchart of steps in a method for performing a localdecode and display procedure for a client, in accordance with oneembodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention relates to improvements for a multi-displayprocessor and wireless peripheral system. While the describedembodiments relate to a host computer system, the same principles andfeatures could be equally applied to host cell phone, host MP3 playerand host media system.

Referring to FIG. 1, the invention provides an architecture for awireless peripheral system 100 to make use of wireless peripherals 130,132 and 136. A host computer 200 processes multiple windows desktopenvironments, typically one for each display, and, besides supportinglocal display 110, produces display update network streams over wirednetwork(s) 290 to each of remote displays 300, 302, 304, etc., or over awireless network 290 to display 306. For a multi-user system, users atthe remote locations are able to time-share the host computer 200 as ifit were their own local computer and have complete support for all typesof graphics, text and video content with the same type of userexperience that could be achieved on a local computer. A host system 200may be connected via a wired connection 138 to a network 128 or may bepart of a wireless network 102. A wired network 128 may be aconventional wired Ethernet network or another wire technology such asCOAX or power line. In any of those cases, the network interface of hostsystem 200 is managed by a Physical (PHY) interface as well as a networkcontroller that is capable of supporting the protocols associated withthe physical interface. In addition to or instead of a wired networkinterface, the host system 200 may be part of a wireless network 102that may conform to elements of the WiFi standards. WiFi is a generalterm encompassing detailed specifications for different versions such as802.11g and 802.11a which operate in different bands, as well as anongoing list of enhancements to each specification including 802.11e, n,s and future derivatives.

As part of a WiFi network, the host system 200 communicates to a localWiFi Access Point 126 which may include a router or may use a separatenetwork router 124. The router may include access to the Internet via aconnection to a Wide Area Network (WAN) or may include a Local AreaNetwork (LAN) connection 128 to an external modem (not shown) thatprovides a WAN connection.

The network router 124 may include a wireless interface or may supportadditional “Remote” WiFi Access Point(s) (APs) 120. The Remote WiFiAccess Point 120 may operate with a different protocol in a different RFband than the local WiFi Access Point 126, may operate within the sameband on a different channel, or may operate on the same channel if thedistance between the access points is sufficient.

The host system 200 may include a variety of local peripherals 112 (akeyboard or mouse) that are connected over specific interfaces such asover a serial port, or a more general wired interface such as aUniversal Serial Bus (USB) port or a 1394 port (not shown). A localdisplay system 110 may be connected over a variety of high speed wiredconnections 222 such as Digital Video Interface (DVI), High DefinitionMultimedia Interface (HDMI) or another electrical or optical interfacefor transmitting uncompressed video data to a display device to the hostsystem 200. The host system may also include interfaces for networkedperipherals that are separate from the primary network interface usedfor the host system to access the Internet. Such an example would be asecond Gigabit Ethernet connection 106 to a peripheral display system138, or an Ultra Wide Band wireless interface 108 to a WirelessPeripheral Device 136.

The host system 200 may also include dedicated wireless links toWireless Peripheral Devices (WPDs) 130 and 132 that may be completelyseparate from the WiFi network or to varying degrees part of the WiFinetwork. Such examples are wireless keyboards, mice and headsets thatmay use either proprietary wireless solutions or may use Bluetooth.Dedicated wireless solutions may be designed for very low poweroperation and may be tuned to the application and for very high batterylife for the wireless peripheral device. Since a WPD such as a keyboardor mouse is low bandwidth, and does not typically need network services,there is a significant advantage to not including a standard WiFiinterface in such a device. Although some host systems may be designedto include proprietary wireless solutions, it is typically necessary toinclude a dongle in the form of an external block connected over a USBport to the host. Including Bluetooth in a system that also includes802.11g WiFi, where both use the same RE band, requires specialattention to avoid RF interference. Bluetooth does include some modes tosupport lower power communication that may be used by Bluetoothperipherals to reduce power consumption. However, in a host systemhaving WiFi, use of Bluetooth still requires an additional wirelesssubsystem.

While a wireless keyboard or mouse is not interesting as part of anetwork when it is not local to the host system, other devices, such asa Voice over IP (VoIP) telephone handset or a display screen may beuseful both within and out of proximity of a host system. To work in astandard system, a WiFi VoIP telephone handset or a WiFi display screenneeds to be part of the WiFi network and communicate directly with arouter or access point. Similarly, if a WiFi keyboard or mouse were tobe designed, the conventional method to communicate with a host for astandard system would be to use the WiFi router or access point as anintermediary between the peripheral and the host. An embodiment of theinvention enhances the WiFi subsystem in the host system, such that ahost system can more directly support wireless peripheral devices in aspecial WiFi peripheral mode (WiFi-P). In some embodiments, the WiFi-Pmode may provide software layers similar to Bluetooth services formanaging host-to-peripheral communication including setup and security.In some embodiments, the communication between the WiFi host system andthe WPD does not use standard IP protocols but instead utilizes a veryefficient layer 2 communication scheme that is based on the MAC addressof the WPD.

WiFi-P Keyboards and Mice

Keyboards and mice are low bandwidth data entry peripherals that lendthemselves to be wireless because they are relatively low power so thatthey can run on batteries, and there is a user benefit in eliminatingthe wires that connect them to the host system. There is relativelylittle need to have a keyboard or mouse enabled as a full networkdevice, so supporting a special peripheral mode where the keyboard andmouse are primarily utilized by a host system does not reduce theirusefulness. In prior art systems, wireless keyboards and mice utilizedeither dedicated wireless links to the computer or utilized Bluetooth,thus requiring additional RF support in the host system.

Since the host system will typically require WiFi for networkconnectivity, it is useful to provide an ability for the host system toutilize WiFi peripherals such as a keyboard and mouse. A specialperipheral mode is required so that the host system can operate instandard infrastructure mode, and maintain connectivity to the networkwhile simultaneously supporting locally connected peripherals. WiFiAd-Hoc mode is the conventional method for WiFi devices to communicatedirectly with each other. Ad-Hoc mode is undesirable in this situation,as a host system while operating in Ad-Hoc mode does not have networkconnectivity. WiFi-P mode allows the host system to maintain networkconnectivity while also supporting a wireless keyboard and mouseutilizing the same WiFi subsystem.

Since both keyboards and mice are low bandwidth devices, thecommunication between these WPDs and the host system may be optimizedaround preserving battery life and assuring that the latency is not toohigh. To preserve the battery life, the RF power levels can be kept low,provided they are high enough to cover the distance between the WPD andthe host system. Other protocol techniques to lower the power arediscussed below. Also, in some embodiments, the protocol can be keptvery simple and does not need to utilize full IP packet basedcommunication. A simpler lower level MAC address based protocol can beutilized.

WiFi-P VoIP Handset

A conventional Voice-over IP (VoIP) handset can be used to initiate andreceive VoIP phone calls whenever such a device is included on a WiFinetwork. The WiFi-P mode may be added to a complete WiFi compatible VoIPhandset, or a special reduced capability handset that includes onlyWiFi-P may be developed. The advantage of the WiFi-P mode is that it isa lower power mode for when the handset is within range of a hostsystem.

WiFi-P Cell Phone Host System

Currently cell phones utilize Bluetooth for the wireless link to a phoneheadset. As cell phones incorporate WiFi for broadband and VoIPcommunication, the same WiFi subsystem can be used to support a WiFi-Pphone headset. The advantage is that only one RF subsystem would berequired in the cell phone. The software services on the cell phonecould be the same as with the Bluetooth services where the management ofthe lower level network and radio could be managed transparently to theapplication.

WiFi-P Audio Headset

MP3 players connect with a host system utilizing either a wired orwireless connection. Some MP3 players have been designed with dockingstations to allow connectivity over a standard network. Increasingly, itwill be desirable to include network connectivity within the MP3 playeritself. For a variety of reasons, that connectivity is likely to be WiFibased. Once the MP3 player includes a WiFi subsystem, it becomespossible to utilize that same WiFi subsystem to support a wireless audioheadset.

Wireless headsets being constrained to be light means using smallerbatteries, making it challenging to satisfy the power requirements forsuch a wireless headset. As such, supporting a Wi-Fi-P mode with lowerpower consumption is desirable.

WiFi-P Display System

Since the cost of a display screen is typically a large component of adisplay system, saving money on the wireless subsystem is not typicallythe first priority. However, achieving high quality for video playbackmay require managing Quality of Service (QoS) which may be betterperformed with a connection from the host to the display that does notnecessarily traverse the entire network. As described below withreference to FIG. 3, a WiFi display system can make use of multiplewireless interfaces to achieve both portability and the highest displayquality based on proximity to the host system and the other wirelessaccess points of a system.

WiFi-P Notebook Wireless Docking Station

Another useful application is a wireless docking station for a notebookcomputer. Notebook computer docking stations often include wiredconnections for a network, keyboard, mouse, external monitor and otherperipherals. Since the network is now often available as a wirelessconnection in an office or home, moving the other devices such askeyboard, mouse and monitor to also utilize wireless technology makesthe concept of a wireless docking station more useful. When a notebookcomputer comes within range, the wireless monitor and keyboard couldquery the user whether he wanted to utilize the notebook. Then, withoutphysically connecting the notebook to any devices, the notebook could beutilized by the wireless monitor and mouse. Additional technologies forconnector-less power charging solutions could be combined so that anotebook, still in a briefcase, could be used in such a wireless dockingstation configuration. A variety of securing mechanisms could also beemployed to prevent unauthorized access to the notebook computer.

WiFi-P for Media Base Station Host System

Current media base stations such as the Sony LF-X1 wireless televisionbase stations utilize both a wired Ethernet and a wireless Ethernetconnection for simultaneously supporting connectivity to the Internetand connectivity to a wireless peripheral “location free” televisiondisplay. The Sony wireless peripheral location free display only workswhen in range of the media base station. Even though the media basestation is part of the network, the media base station and the locationfree television display can not automatically switch to a network basedconnection when the location free television is out of range of themedia base station. Only after the user manually configures the setupcan the display access content from the base station over anetwork-based connection. The preferred operation of the display is towirelessly connect directly to the wireless media base station when itis in range, and when it is not in range to automatically switch to anetwork based connection. In a preferred embodiment of the presentinvention, when a wireless display is out of wireless range of a mediabase station, but within range of another remote access point orconnected to the network infrastructure via another means, the mediabase station uses the network infrastructure to provide the requestedmedia to the wireless peripheral. The network infrastructure for such apath may require use of both wired and wireless links and as such themedia base station may use the wired network infrastructure and act asthe host to the wireless display whether the wireless display is withinrange of the media base station's WiFi connection or whether the hostingis performed over the network infrastructure.

The current media base station is required to utilize a wired Ethernetconnection for access to the LAN. The WiFi subsystem can only be usedfor directly supporting the wireless display or where the media basestation acts as an Access Point for other wireless devices. The wirelessmedia base station is not able to utilize the WiFi subsystem forconcurrent connectivity to the Internet and connectivity to the locationfree TV because it lacks support for a WiFi-P mode that allowssimultaneous Internet access and support for a wireless peripheral.Adding WiFi-P mode to the wireless media base station would allow fullwireless support where the media base station connects wirelessly toboth the wireless location free display and the wireless infrastructure.

WiFi-P for Sensors

Current sensors are designed to support different RE sensor networks buttypically not WiFi. For areas that have WiFi coverage, designing sensorsto support WiFi-P mode may be a good approach. In this case, each sensordoes not need to fully support the complete WiFi network stack, but mayrespond to or initiate communication with a WiFi based host system. Inother sensor systems, the sensors may wish to be part of a secure WiFinetwork which requires the sensors to acquire a network key.

WiFi-P Authentication

An additional use for WIN Peripheral Mode is as a means to authenticatean enrollee onto a secure WiFi network. A typical issue for securenetworks is how to authenticate a new enrollee without creating apotential security hole that could be exploited to compromise thenetwork. System security typically dictates that the authenticationneeds to occur prior to the enrollee having any access to the network.Without having access to the network, the authentication of a newenrollee needs to be performed “out of band”—meaning without use of thesecure network. Typically, the authentication requires providing theenrollee with a key that will allow access to the network. Various priorart methods have been proposed for such a scheme including using USB fortransferring the key, or requiring some type of user activation bymanually coding in a pre-shared key such as with WiFi Protected AccessPre-Shared Key (WPA PSK) Mode. Some existing schemes involve the use ofmore than one key, where the first key gives limited access onto thenetwork, and a second key is used as a unique identifier for theenrollee. Other multi-key schemes (such as Temporal Key IntegrityProtocol (TKIP)) may also be used. The prior art means for initial keyexchange, whether via USB, via manual keying or by using another out ofband connection, adds user complications, may open security holes andmay increase the system costs.

In one embodiment of this invention, WiFi. Peripheral Mode can be usedbetween the host and the enrollee as an out of band channel for thepassing of key information for validating first time WiFi enrollees. Thehost system is first used to validate an enrollee. Then the host systemrequests a WiFi network key on behalf of the enrollee. The host systemcommunicates with the enrollee using the WiFi RF subsystems of both thehost and enrollee, but the initial providing of the key utilizes theWiFi Peripheral Mode and is “out of band.” In order to maintain propernetwork security, the host system needs to validate the enrollee priorto providing the key. Once the host system provides the enrollee withthe network key, the enrollee becomes capable of either operating viathe host system in WiFi-P mode, or the enrollee becomes capable ofutilizing the provided network key to independently operate on thesecure WiFi network. In another embodiment, WiFi-P mode is used inconjunction with TKIP or with Robust Secure Network (RSN) procedures toallow out of band passing of key information.

As an example of an embodiment of this invention, a temperature sensormay be added as an enrollee to a secure WiFi network. The temperaturesensor has a serial number but has no user interface and no inputcontrols (except possibly an on-off switch and a reset means). Thetemperature sensor is turned on and its beacon is recognized by a hostsystem. The beacon utilizes the WiFi RF subsystem, but since the sensordoes not have a network key, the host system can only communicate withthe sensor using an out of band WiFi RF based protocol. A user of thehost system, where the host system may be a computer or some simplifiedmanagement terminal with a display and simple I/O, validates that theenrollee beacon corresponds to the serial number of the physical sensordevice. Once validated, the user initiates a request via the host systemfor the network to validate the new enrollee. The network provides thekey for the enrollee to the host system via the secure WiFi channel. Thehost system then provides the key to the enrollee via the out of bandWiFi peripheral channel. Once the enrollee receives the key, theenrollee may utilize the WiFi RF subsystem to communicate over thesecure WiFi network.

WiFi-P Network Modes

As mentioned, WiFi Ad-Hoc mode is the conventional method for WiFidevices to communicate directly with each other. Ad-Hoc mode isundesirable in this situation since a host system in Ad-Hoc mode doesnot have network connectivity. WiFi-P mode allows the host system tomaintain network connectivity while also supporting a wireless keyboardand mouse utilizing the same WiFi subsystem. When a host system operatesin WiFi-P mode, access points that are unaware of the WiFi-P mode simplyview the host system as operating in standard infrastructure mode. Inthe case of peripherals to the host system, the router and networkinfrastructure may be completely unaware of the peripheral devices, mayview the peripheral devices as an extension of the host systemcapability, or may view the host system as providing Access Pointcapabilities to the peripherals.

Where the peripherals are designed to only communicate with the hostsystem, the router and network infrastructure do not need to know aboutthe peripherals and no services need to be provided to the peripherals.The communication between the peripherals and the host system operatesin the WiFi-P mode which is effectively ignored by the network. Wherethe peripherals may both operate in direct communication with the hostsystem and operate on the network itself, the router may provide basicservices at initialization time of the device. For example, a DynamicHost Configuration Protocol (DHCP) server may provide the peripheraldevice with an IP address for use on the network.

WiFi-P Protocols

As the WiFi specifications continue to evolve with new additions,capabilities for the WiFi host system and the WiFi infrastructuredevices will continue to improve. Advances such as 802.11e for QoS and802.11s for mesh support will enhance network capabilities. The WiFi-Pmode also is designed to utilize ongoing advancements made in WiFiprotocols. The WiFi-P mode can operate in any WiFi environment, althoughwhen there are fewer protocol options for the network, there may be lessefficiency for the host-to-WPD communication.

In the simplest embodiment, the WPD includes a complete WiFi protocolstack and performs the required WiFi arbitration. In such a case, theAP, host system and WPD all need to run an interoperable WiFi protocol.The direct communication between the host system and the wirelessperipheral utilizes the standard higher layer protocols that are part ofthe WiFi specification. Based on the distance between the WPD and thehost system, the active power for the RF subsystem may be reduced toshorten the range. The additional optimizations for the communicationbetween the peripheral device and the host system are to reduce thecomplexity of the WPD, reduce the power requirements of the WPD, and/orimprove the QoS.

Even though the WPD communicates with the host system and not directlywith the wireless network, all of the communication may be using thesame or an overlapping WiFi RF channel. Even at lower active RF powerlevels, the chance exists for interference from using the same or anoverlapping RF channel. Also, the host system may maintain itscommunication channel with the wireless network so it should not go“offline” while communicating with the WPD. With these constraints, thegeneral methods to reduce the complexity of the wireless peripheralrequire that the host system perform the WiFi protocol arbitration sothat the RF channel is clear for the wireless peripheral and host tocommunicate. With the host system managing the WiFi protocolarbitration, the WPD can essentially be in sleep mode until such time asit is in operation. The wake out of sleep mode can be initiated byeither the WPD or the host system.

A general WiFi network controller 228 consists of an RF subsystem, abaseband subsystem (sometimes referred to as layer 2) and a MACsubsystem (sometimes referred to as layer 3). These layer references arenot technically rigid. Other terms such as the PHY layer sometimes referto both layer 1 and layer 2, and the MAC layer often includes functionsof higher layers 4 and 5. In some systems utilizing existingtechnologies, albeit at a higher cost of implementation, the host systemmay have two non-overlapping RF channels active of which one can be usedfor accessing the WiFi network and the other can be used for accessingthe WiFi peripheral devices. Depending on the controls available withinthe host system for the two channels, access to the two channels may beconcurrently managed with different baseband controllers, or there maybe some limitations to the ability for each channel to be usedconcurrently. A host system could also include two complete WiFi networkcontrollers including the MAC where each operates on an differentnon-overlapping RF channel.

When the host system and the WPD communicate, since the host system hasarbitrated for and controls the WiFi RF channel, the specificcommunication between the two can be very efficient. In a preferredembodiment, the WPD and host system use a very efficient protocol thatoperates below the conventional IP layer. This is sometimes called alayer 2 protocol. Instead of being based on IP addresses which are partof the network, such layer 2 protocols may be based on the unique MACaddresses of the devices. There are many ways to achieve the goal ofefficient communications between the host system and WPDs and thepreferred embodiments described here are just a few of the possibilitiescovered by the present invention.

For standardized communication between WiFi devices, the base WiFi MACstandard includes two channel access coordination functions. The simplerone is called Distributed Coordination Function (DCF) and the morecomplicated one is called Point Coordination Function (PCF). In the caseof DCF, there is the least flexibility for the base

The PCF mode has a more robust mechanism using Contention Free Periods(CFPs) during which devices can be polled. During reserved timeslotswithin CFPs, the host system can more assuredly poll the WPD to eitherinitiate a request or see if there are pending requests from the WPD.The PCF mode is an optional part of the 802.11g standard although with802.11e there is a mandatory Hybrid Coordination Function (HCF) thatprovides a contention free polling mechanism much like the PCF mode. Inall cases, the host system can only utilize the WiFi protocols that areavailable between the host system and the AP for gaining control of thechannel although once the host system has the channel, the communicationwith the WPD may be more specific to the two devices.

FIG. 2 is a block diagram of a host computer 200 for a wirelessperipheral system 100 in accordance with an embodiment of the invention.The basic components of host computer 200 preferably include, but arenot limited to, a CPU subsystem 202, a bus bridge-controller 204, a mainsystem bus 206 such as PCI express, local I/O 208, main RAM 210, and agraphics and display controller 212 having one or more dedicated outputpaths SDVO1 214 and SDVO2 216, and possibly its own memory 218. Thegraphics and display controller 212 may have an interface 220 thatallows for local connection 222 to a local display 110. Host computer200 also preferably includes a multi-display processor 224 that has bothinput paths SDVO1 214 and SDVO2 216 from the graphics and displaycontroller 212 and an output path 226 to network controller 228. Inaddition to or instead of dedicated path 226, Multi-Display Processor224 may be connected by the main system bus 206 to the NetworkController 228. The multi-display processor 224 may include a dedicatedRAM 230 or may share main system RAM 210, graphics and displaycontroller RAM 218 or network controller RAM 232. Those familiar withcontemporary computer systems will recognize that the main RAM 210 maybe associated more closely with the CPU subsystem 202 as shown at RAM234. Alternatively the RAM 218 associated with the graphics and displaycontroller 212 may be unnecessary as the host system 200 may share themain RAM 210.

The function of multi-display processor 224 is to receive one or moredisplay refresh streams over each of SDVO1 214 and SDVO2 216, manage theindividual display outputs, process the individual display outputs,track which portions of each display change on a frame-by-fame basis,encode the changes for each display, format and process what changes arenecessary and then provide a display update stream to the networkcontroller 228. The path from the multi-display processor 224 to thenetwork controller 228 may be a direct connection 226 or a connectionutilizing the main system bus 206, or the multi-display processor mayfeed the output back to an input port on the Graphics and DisplayController 212. In some configurations, the multi-display processor 224does not need a connection to the main system bus 206 and all of itscommunications can be managed through the interfaces with the Graphicsand Display Controller 212. The Graphics and Display Controller 212 mayinclude a control bus (not shown) that can be used for programming theMulti-Display Processor 224 or the control information may be includedas special control packets on the data transfer busses. The return pathfrom the Multi-Display Processor 224 to the Graphics and DisplayController 212 may be part of a control path or may be another type ofdata port such as a parallel data input port, a serial data input portor some standard port such as an enhanced speed version of ConsultativeCommittee on International Radio Standard 556 (CCIR656) or Digital VideoBroadcast (DVB). FIG. 8B shows such a configuration where a DisplayEncoder IC 803 utilizes control and data bus 804 and does not include aconnection to the main system bus 206.

Network and RF controller 228 processes the display update stream andprovides the network communication over one or more network connections290 to the various display devices. These network connections can bewired or wireless. In the case of wireless networks, a single network ora combination of wireless networks may be used. For example, hostcomputer 200 may use UWB to wirelessly provide display data to one ormore high resolution displays and may use WiFi to communicate with otherperipherals.

The implementation and functionality of a multi-display system 100 arefurther discussed below in conjunction with FIGS. 3 through 10.

FIG. 3 is a block diagram of a remote display system 300, in accordancewith one embodiment of the invention, which preferably includes, but isnot limited to, a display screen 310, a local RAM 312, and a remotedisplay system controller 314. The remote display system controller 314includes an I/O control subsystem 316 which has connections 318, 320 and322 for devices that in some installations due to proximity do notconnect directly to the host system. Such devices may include speakersfor reproducing audio or a Universal Serial Bus (USB) connection whichcan support a variety of devices. The connections can be dedicatedsingle purpose such as a PS/2 style keyboard or mouse connection, ormore general purpose such as a USB. In another embodiment, the I/O couldsupport a game controller, a local wireless connection, an IR connectionor no connection at all. Remote display system 300 may also includeother peripheral devices such as a DVD drive (not shown).

Also shown in FIG. 3 is an optional Broadcast Antenna, Tuner andDemodulation 342 subsystem with a connection 344 to an antenna (notshown). As a display device, it may be desirable for controller 314 toinclude the ability to directly receive broadcasts such as audio andvideo programming. The sources for such broadcasts may include AmplitudeModulation (AM) Radio, Frequency Modulation (FM) radio, digital radio,satellite radio, national Television Systems Committee (NTSC)television, Advanced Television Systems Committee (ATSC) Television,DVB-H or another form of Digital Video Broadcasting (DVB) television,cellular network based broadcasts, or a variety of other over the airbroadcast transmissions. The information as to what is being tuned bythis subsystem is made available to the CPU 324 and the software withinthe remote display system 300. This information can be relayed via thenetwork controller 326 back to the host system 200. Some uses for such arelay are coordination of viewing preferences.

For example, a user may request to record a television broadcast basedon viewing a DVB-H broadcast. Since the remote display system 300 doesnot include a record capability, that request is forwarded to the hostsystem 200 to record the content. This can be done in a variety of waysincluding having the remote display system 300 stream the content to thehost system 200 for recording, or the host system may record the contentfrom another more local

Some embodiments of the invention do not require any inputs at theremote display system 300. An example of such a system is a retail storeor an electronic billboard where different displays are available atdifferent locations and can show variety of informative and entertaininginformation. Each display can be operated independently and can beupdated based upon a variety of factors. A similar system could alsoinclude some displays that accept touch screen inputs as part of thedisplay screen, such as an information kiosk.

The remote display system may function as either a second display or asthe primary display for a laptop PC. The laptop PC input may employ thekeyboard and mouse that are an integrated part of the laptop PC, or anexternal keyboard and mouse that connect via USB, a proprietary RFsolution, Bluetooth or WiFi. In the case of WiFi keyboards and mice, theWiFi peripheral mode described with reference to FIG. 1 can be used toallow the keyboard and mice to operate on batteries in a low power mode.Conserving battery power is a function of several factors including theactive power usage while in operation, the standby power usage while notcurrently performing transfers, and a sleep mode while the device is inan inactive state.

A variety of network connectivity options may be supported by a remotedisplay system 300. A common network example is Ethernet, such as CAT 5wiring running some type of Ethernet, preferably gigabit Ethernet, wherethe I/O control path may use an Ethernet supported protocol such asstandard Transport Control Protocol and Internet Protocol (TCP/IP), someform of lightweight handshaking in combination with User DatagramProtocol (UDP) transmissions, or a lower layer level 2 based MACprotocol. Also, a variety of wireless networking options may besupported. Referring back to FIG. 1, and considering the device labeledWireless Peripheral 136 to be the remote display system 300 there may bea wireless path 108 directly to the host system 200, a wireless path 102that utilizes WiFi Access Point 126, or wireless path 104 which connectsto a “Remote” WiFi Access Point 120.

The wireless path 108 from the host system 200 to the wirelessperipheral device 136 may be a standard WiFi link where the host system200 acts as a link in a mesh or as an access point, it may be adedicated wireless link which may utilize a separate WiFi channel or adifferent wireless standard such as Ultra Wide Band (UWB), or it may bea wireless link utilizing the WiFi-P mode. The wireless link,communication protocol and the video CODEC used can be dynamicallydetermined based on the connections available. For example, while inclose range, the host system 200 and the wireless peripheral device 136may utilize UWB using a CODEC that matches the compression to thebandwidth available on the link. If the user moves the wirelessperipheral device 136 out of range of the UWB connection, the wirelesslink may switch over to WiFi-P mode where the CODEC may requireadditional compression to fit the video transmission into the availablebandwidth. The WiFi-P mode allows for some added QoS controls that maynot be available when the WiFi access point 126 is included in thetransfers. However, if the wireless peripheral device 136 is later outof range of the host system 200, but is still in range of the remote WINaccess point 120, then the communication link may use network protocolsthat are supported through the network infrastructure.

In each case, if the wireless peripheral device 136 includes a display,the network carries the encoded display data required for the displaywhere the data decoder and frame manager 328 and the display controller330 are used to support all types of visual data representations thatmay be rendered at the host computer 200 and display them locally ondisplay screen 310.

The display controller 330, data decoder and frame manager 328, and CPU324 work together to manage a representation of the current image framein the RAM 312 and to display the image on the display 310. Typically,the image will be stored in RAM 312 in a format ready for display, butin systems where the RAM cost is an issue, the image can be stored inthe encoded format. When stored in an encoded format, in some systems,the external RAM 312 may be replaced by large buffers within the remotedisplay system controller 314.

After the display is first initialized, the host system 200 provides afull frame of data for decode and display over the network. Followingthat first frame of display data, the host system 200 need only sendpartial frame information over the network 290 as part of the displayupdate network stream. If none of the pixels of a display are changedfrom the prior frame, the display controller 330 can refresh the displayscreen 310 with the prior frame contents from local storage 328. Whenpartial frame updates are sent in the display update network stream, CPU324 and display data decoder 328 perform the necessary processing stepsto decode the image data and update the appropriate area of RAM 312 withthe new image. During the next refresh cycle, the display controller 330will use this updated frame for display screen 310. The processing forconversion and storage of the display update network stream is describedin further detail with reference to FIGS. 4 through 10 below.

An embodiment of host computer 200 may support multiple remote displaysystems 300, each of which may include a display system, utilizing whatis conventionally associated with a single graphics and displaycontroller 212 and a single SDVO connection. The method of multi-userand multi-display management is represented in FIG. 4 by RAM 218 dataflowing through path 402 and the display controller-1 portion 404 of thegraphics and display controller 212 to the output connections SDVO1 214and SDVO2 216.

For illustration purposes, RAM 218 is shown organized into varioussurfaces each containing display data for multiple displays. The primarysurfaces 406, Display 1 through Display 12, are illustrated with aprimary surface resolution that happens to match the display resolutionfor each display. This is for illustrative purposes although there is norequirement for the display resolution to be the same resolution as thatof the primary surface. The other area 408 of RAM 218 is showncontaining secondary surfaces for each display and supporting off-screenmemory. The RAM 218 will typically be a common memory subsystem fordisplay and graphics subsystem 212, although the RAM may also be sharedwith main system memory 210 or with the memory of another processor inthe system 100. In a shared memory system, contention may be reduced ifthere are available multiple concurrent memory channels for accessingthe memory. The path 402 from RAM 218 to the graphics and displaycontroller 212 may be time-shared.

The graphics and display controller 212's 2D, 3D and video graphicsprocessors 410 are preferably utilized to achieve high graphics andvideo performance. The graphics processing units may include 2Dgraphics, 3D graphics, video encoding, video decoding, scaling, videoprocessing and other advanced pixel processing. The display controllers404 and 412 may also include processing units for performing functionssuch as blending and keying of video and graphics data, as well asoverall screen refresh operations. In addition to the RAM 218 used forthe primary and secondary display surfaces, there is sufficientoff-screen memory to support various 3D and video operations. Thedisplay controllers will typically combine the primary surface with oneor more secondary surfaces to support the display output of a compositeframe, although it is also possible for the graphics processors 410 toperform the compositing into a single primary surface.

In a single-display system, display controller-1 404 would be configuredto access RAM 218, process the data and output a proper displayresolution and configuration over output SDVO1 214 for a single displaydevice. Preferably, the display controller-1 404 is configured for adisplay size that is much larger than a single display to therebyaccommodate multiple displays. Assuming the display controller-1 404 ofa typical graphics and display controller 212 was not specificallydesigned for a multi-display system, the display controller-1 404 cantypically only be configured for one display output configuration at atime. It may however be practical to consider display controller-1 404to be configured to support an oversized single display as that is oftena feature used by “pan and scan” display systems and may be just afunction of setting the counters in the display control hardware.

In the illustration of FIG. 4, consider that each display primarysurface represents a 1024×768 primary surface corresponding to a1024×768 display. Stitching together six 1024×768 displays, three acrossand two down, would require display controller 212 to be configured tothree times 1024, or 3072 pixels of width, by two times 768, or 1536pixels of height. Such a configuration would accommodate Displays 1through 6.

Display controller-1 404 would treat the six tiled displays as one largedisplay and provide the scan line based output to SDVO1 output 214 tothe multi-display processor 224. Where desired, display controller-1 404would combine the primary and secondary surfaces for each of the sixtiled displays as one large display. The displays labeled 7 through 12could similarly be configured as one large display for displaycontroller-2 412 through which they would be transferred over SDVO2 216to the multi-display processor 224.

In a proper configuration, FIG. 6 multi-display processor 224 managesthe six simultaneous displays property and processes as necessary todemultiplex and capture the six simultaneous displays as they arereceived over SDVO1 214.

In the FIG. 4 primary surface 406, the effective scan line is threetimes the minimum tiled display width, making on-the-fly scan line basedprocessing considerably more expensive. In a preferred environment foron-the-fly scan line based processing, the display controller-1 404 isconfigured to effectively stack the six displays vertically in one planeand treat the tiled display as a display of resolution 1024 pixelshorizontally by six times 768, or 4608, pixels vertically. To the extentit is possible with the flexibility of the graphics subsystem, it isbest to configure the tiled display in this vertical fashion tofacilitate scan line based processing. Where it is not possible toconfigure such a vertical stacking, and instead a horizontal orientationneeds to be included, it may be necessary to only support precinct basedprocessing where on-the-fly encoding is not done. In order to minimizelatency, when the minimum number of lines has been scanned, the precinctbased processing can begin and effectively be pipelined with additionalscan line inputs.

FIG. 5 shows a second configuration where the tiled display is set to1600 pixels horizontally and two times 1200 pixels or 2400 pixelsvertically. Such a configuration would be able to support two remotedisplay systems 300 of resolution 1600×1200 or eight remote displays of800×600 or a combination of one 1600×1200 and four 800×600 displays.FIG. 5 shows the top half of memory 218 divided into four 800×600displays labeled 520, 522, 524 and 526.

Additionally, the lower 1600×1200 area could be sub-divided to anarbitrary display size smaller than 1600×1200. As delineated withrectangle sides 530 and 540, a resolution of 1280×1024 can be supportedwithin a single 1600×1200 window size. Because the display controller-1404 is treating the display map as a single display, the full rectangleof 1600×2400 would be output and it would be the function of themulti-display controller 224 to properly process a sub-window size forproducing the display output stream for the remote display system(s)such as system 300. A typical high quality display mode would beconfigured for a bit depth of 24 bits per pixel, although often theconfiguration may utilize 32 bits per pixel as organized in RAM 218 foreasier alignment and potential use of the extra eight bits for otherpurposes when the display is accessed by the graphics and videoprocessors. One example use of the extra bits may include hardwareassisted tracking of which pixels or blocks of pixels have been changed.

FIG. 5 also illustrates the arbitrary placement of a display window 550in the 1280×1024 display. The dashed lines 546 of the 1280×1024 displaycorrespond to the precinct boundaries assuming 128×128 precincts. Whilein this example the precinct edges line up with the resolution of thedisplay mode, such alignment is not necessary. As is apparent fromdisplay window 550 the alignment of the display window boundaries doesnot line up with the precinct boundaries. This is a typical situation asa user will arbitrarily size and position a window on a display screen.In order to support remote screen updates that do not require the entireframe to be updated, all of the precincts that are affected by thedisplay window 550 need to be updated. Furthermore, the data type withinthe display window 550 and the surrounding display pixels may be ofcompletely different types and not correlated. As such, the precinctbased encoding algorithm, if it is lossy, needs to assure that there areno visual artifacts associated with either the edges of the precincts orwith the borders of the display window 550. The actual encoding processmay occur on blocks that are smaller, such as 16×16, than the precincts.

For wavelet based encoding schemes, the process step most likely tocause visual differences between precincts or tiles for different framesof data is the Discrete Wavelet Transform (DWT) itself. One approach toavoid the visual differences is to perform the DWT on a larger tile oron the entire image for each frame. As will be explained in detailbelow, the DWT results for each precinct of each frame can be used togenerate the signature for each precinct to determine if that precinctneeds to be re-encoded. If the precinct signature has not changed fromthe prior frame, then no re-encoding is necessary. For precincts wherethe signature has changed, the precinct data is re-encoded. Comparingsignatures after the DWT has an added benefit in the case where even ifthe frame data has not changed, but other parameters such as thecompression ratio that would affect the encoded image have changed, thepost DWT signature will have changed indicating that the data for thatprecinct, tile or frame needs to be resent. Of course there are avariety of conditions where the frame can be reset so that the entireframe image is updated.

The illustration of the tiled memory is conceptual in nature as a viewfrom the display controller-1 404 and display controller-2 412. Theactual RAM addressing will also relate to the memory page sizes andother considerations. Also, as mentioned, the memory organization is nota single surface of memory, but multiple surfaces, typically includingan RBG surface for graphics, a YUV surface for video, and an area ofdouble buffered RGB surfaces for 3D. The display controller combines theappropriate information from each of the surfaces to composite a singleimage. The compositing may also include alpha blending, transparency,color keying, overlay and other similar functions to combine the datafrom the different planes. In Microsoft Windows XP® terminology, thedisplay can be made up of a primary surface and any number of secondarysurfaces. The FIG. 4 sections labeled Display 1 through Display 12 canbe thought of as primary surfaces 406 whereas the secondary surfaces 408are managed in the other areas of memory. Surfaces are also sometimesreferred to as planes.

The 2D, 3D and video graphics processors 410 would control each of thesix displays independently with each possibly utilizing a windowed userenvironment in response to the display requests from each remote displaysystem 300. This could be done by having the graphics and videooperations performed directly into the primary and secondary surfaces,where the display controller-1 404 composites the surfaces into a singleimage. Another example is to primarily use the primary surfaces and toperform transfers from the secondary surfaces to the primary surfaces.As long as the transfers are coordinated to occur at the right times,adverse display conditions associated with non-double buffered displayscan be minimized. The operating system and driver software may allowsome of the more advanced operations for combining primary and secondarysurfaces to not be supported by indicating to the software that suchadvanced functions, such as transparency, are not available.

In a typical prior art system, a display controller-1 404 would beconfigured to produce a refresh rate corresponding to the refresh rateof a local display. A typical refresh rate may be between 60 and 85 Hzalthough possibly higher and is somewhat dependent on the type ofdisplay and the phosphor or response time of the physical displayelements within the display. Because the graphics and display controller212 is split over a network from the actual display device 310, screenrefreshing needs to be considered for this system partitioning.

Considering the practical limitations of the SDVO outputs from anelectrical standpoint, a 1600×1200×24 configuration at 76 Hz isapproximately a 3.5 Gigabits per second data rate. Increasing the tileddisplay to two times the height would effectively double the data andwould require cutting the refresh rate in half to 38 Hz to still fit ina similar 3.5 Gigabits per second data rate. Because in thisconfiguration the SDVO output is not directly driving the displaydevice, the refresh requirements of the physical display elements of thedisplay devices are of no concern. The refresh requirements can insteadbe met by the display controller 330 of the remote display controller314.

Though not related to the refresh rate, the display output rate for thetiled display configuration is relevant to the maximum frame rate of newunique frames that can be supported and it is one of the factorscontributing to the overall system latency. Since full motion is oftenconsidered to be 30 frames per second, the example configurationdiscussed here at 36 Hz could perform well with regard to frame rate.Other lower frame rates can also be supported even though they are notfull motion. In general, the graphics and video drawing operations thatwrite data into the frame buffer are not aware of the refresh rate atwhich the display controller is operating. Said another way, the refreshrate is software transparent to the graphics and video drawingoperations.

Each display refresh stream output on SDVO1 214 includes data toindicate which display is the target recipient of the update and wherewithin the display (which precincts for systems that are precinct based)the new updated data is intended for, and includes the encoded data forthe display.

Note that in FIG. 5, the window indicated by box 550 does not align withthe drawn precincts and may or may not align with blocks of ablock-based encoding scheme. Some encoding schemes will allow arbitrarypixel boundaries for an encoding subframe. For example, if the window550 utilizes text and the encoding scheme utilized RLE encoding, theframe manager can set the sub-frame parameters for the window to beencoded to exactly the size of the window. When the encoded data is sentto the remote display system, it will also include both the window sizeand a window origin so that the data decoder and frame manager 328 candetermine where to place the decoded data into a decoded frame.

If the encoding system used does not allow for arbitrary pixelalignment, then the pixels that extend beyond the highest block sizeboundary either need to be handled with a pixel-based encoding scheme,or the sub-frame size can be extended beyond the window 550 size. Thesub-frame size should only be extended if the block boundary will not beevident by separately compressing the blocks that extend beyond thewindow.

Assuming window 550 is generated by a secondary surface overlay, thesoftware tracking layer can be useful for determining when changes aremade to subsequent frames. Even though the location of the secondarysurface is known, because of various overlay and keying possibilities,the data to be encoded should come from a/the stage after the overlayand keying steps are performed by either one of the graphics engines orby the display processor.

FIG. 6 is a block diagram of the multi-display processor subsystem 600which includes the multi-display processor 224 and the RAM 230 and otherconnections 206, 214, 216 and 226 from FIG. 2. The representative unitswithin the multi-display processor 224 include a frame comparer 602, aframe manager 604, a data encoder 606, and system controller 608. Thesefunctional units are representative of the processing steps performedand could be performed by a multi-purpose programmable solution, a DSPor some other type of processing hardware.

Though the preferred embodiment is for multiple displays, for the sakeof clarity, this disclosure will first describe a system with a singleremote display screen 310. For this sample remote display, the remotedisplay system 300, the graphics and display controller 212 and themulti-display processor 224 are all configured to support a commondisplay format typically defined as a color depth and resolution.Configuration is performed by a combination of existing and enhancedprotocols and standards including Digital Display Control (DDC), andUniversal Plug and Play (uPNP), and utilizing the multi-display supportwithin the Windows® or Linux® operating systems, and may be enhanced byhaving a management setup and control system application. Other newerstandards such as Display Port and Unified Display Interface (UDI) haveadditional protocols to query and set up the display device.

The graphics and display controller 212 provides the initial displaydata frame over SDVO1 214 to the multi-display processor 224 where theframe manager 604 stores the data over path 610 into the user displayportion 612 of memory 230. Frame manager 604 keeps track of the displayand storage format information for the frames of display data. When thesubsequent frames of display data are provided over SDVO1 214, the framecomparer 602 compares the subsequent frame data to the just prior framedata already stored in RAM 230. The prior frame data is read from RAMover path 610. The new frame of data may either be compared as it comesinto the system on path 214, or may be first stored to memory by theframe manager 604 and then read by the frame comparer 602. Performingthe comparison as the data comes in saves the memory bandwidth of anadditional write and read to memory and may be preferred for systemswhere memory bandwidth is an issue. This real time processing isreferred to as “on the fly” and may be a preferred solution for reducedlatency.

The frame compare step identifies which pixels and regions of pixelshave been modified from one frame to the next. The comparison of framesis performed by either comparing the actual pixels or by comparing asignature of a region of pixels and the tracking of the changes from oneframe to the next is typically performed at the higher regiongranularity. This higher granularity makes the management of the framedifferences more efficient. In one embodiment, a fixed grid of 128×128pixels, referred to as a precinct, may be used for tracking changes fromone frame to the next. In other systems the precinct size may be largeror smaller, and instead of square precincts, the tracking can also bedone on the basis of a rectangular region, scan line or a group of scanlines. The block granularity used for compression may be a differentsize than the precinct and they are somewhat independent although theminimum precinct size would not likely be smaller than the block size.

The frame manager 604 tracks and records which precincts or groups ofscan lines of the incoming frame contain new information and stores thenew frame information in RAM 230, where it may replace the prior frameinformation and as such will become the new version of prior frameinformation. Thus, each new frame of information is compared with theprior frame information by frame comparer 602. The frame manager 604also indicates to the data encoder 606 and to the system controller 608when there is new data in some of the precincts and which ones thoseprecincts are. From an implementation detail, the new data may bedouble-buffered to assure that data encoder 606 accesses are consistentand predictable. In another embodiment where frames are compared on thefly, the data encoder may also compress data on the fly. This isparticularly useful for scan line and multi-scan line based datacompression.

For block based data encoding the data encoder 606 accesses the modifiedprecincts of data from RAM 230, or large internal buffers, andcompresses the data. System controller 608 keeps track of the displayposition of the precincts of encoded data and manages the data encodingsuch that a display update stream of information can be provided via themain system bus 206 or path 226 to the network controller. Because theprecincts may not correspond to any particular display surface, in thepreferred embodiment any precinct can be independently encoded withoutconcern for creating visual artifacts between precincts or on the edgesof the precincts. However, depending on the type of data encoding used,the data encoder 606 may require accessing data beyond the changedprecincts in order to perform the encoding steps. Therefore, in order toperform the processing steps of data encoding, the data encoder 606 mayaccess data beyond just the precincts that have changed. Losslessencoding systems should not have a problem with precinct edges.

A further enhanced system does not need to store the full prior frame inorder to compare on-the-fly. Consider a system that includes eight linebuffers for the incoming data and contains storage for a signatureassociated with each eight lines of the display from the prior frame. Asignature is a calculated number that is generated through some hashingof a group of data. While the original data can not be reconstructedfrom the signature, the same input data will always generate the samesignature, whereas any change to any of the pixels of the input datawill generate a different signature. Using 20 bits for a signature givestwo raised to the twentieth power, or about one million, differentsignature possibilities. This means there would be about a one in amillion chance of an incorrect match. The number of bits for thesignature can be extended further if so desired.

The signature may be generated in conjunction with or separate from thepath used for the encoding. As mentioned above, the signature generationmay also be performed in conjunction with one of the steps of theencoding process such as using the DWT output for signature generation.In one implementation, each scan line is encoded on the fly using theprior seven incoming scan lines and the data along the scan line asrequired by the encoding algorithm. As each group of eight scan lines isreceived, the signature for that group is generated and compared to thesignature of those same eight lines from the prior frame. If thesignature of the new group of eight scan lines matches the signature ofthe prior frame's group of eight scan lines, then it can be safelyassumed that there has been no change in display data for that group ofscan lines, and the system controller 608 can effectively abort theencoding and generation and transmission of the display update streamfor that group of scan lines. If after receiving the eight scan lines,the signatures for the current frame and the prior frame are different,then that block of scan lines contains new display data and systemcontroller 608 will encode the data and generate the display updatestream information for use by the network controller 228 in providingdata for the new frame of a remote display.

The signature generation can be done with minimal buffering of the datawhere each pixel, or group of pixels, contributes to the signature andis no longer needed for the signature operation. However, the pixel datamay still be needed if the pixels need to be encoded. Since the resultof the signature will not be known until after the signature has beencompleted on the complete block of data, the incoming pixels cannot bediscarded until the result of the signature comparison is known. Thepixel data can be stored in buffer memory either on-chip or off-chip. Inone optimization to conserve memory bandwidth, the pixels can bebuffered on-chip and only if the signature does not match, then thepixels are transferred to the off-chip memory. The internal memory maybe double buffered so that while the first buffer is being transferredto main memory for encoding, the second buffer is receiving the nextgroup of pixels for the next signature operation.

In order to improve the latency, the encoding and signature generationand comparison may be partially overlapped or done in parallel. In onesystem the encoding of the data is performed from the pixels stored inan internal buffer memory. Such a system may use double or multi-waybuffering with either line buffers or with another memory arrangement.As mentioned, one of the initial encoding processing steps may be usedas part of the process for signature generation. If the original datawill not be required for the full encoding, the resulting output of thatinitial processing step may be stored instead of the initial data.

When the prior frame data is not used in the encoding, the encoding stepuses one of any number of existing or enhanced versions of known lossyor lossless two dimensional compression algorithms, including but notlimited to Run Length Encoding (RLE), Wavelet Transforms, DiscreteCosine Transform (DCT), Vector Quantization (VQ) and Huffman Encoding.Different types of content benefit to different extents based on thechosen encoding scheme. For example, frames of video images containvarying colors but not a lot of sharp edges, which is fine for DCT basedencoding schemes, whereas text includes a lot of white space betweencolor changes but has very sharp edge transitions that need to bemaintained for accurate representation of the original image and thusDCT would not be the most efficient encoding scheme. The amount ofcompression required will also vary based on various system conditionssuch as the network bandwidth available and the resolution of thedisplay.

For systems that include the prior frame data as part of the encodingprocess, more sophisticated three dimensional compression techniques canbe used where the third dimension is the time domain of multiple frames.Such enhancements for time processing include various block matching andblock motion techniques which can differ in the matching criteria,search organization and block size determination.

While the discussion of FIG. 6 primarily described the method forencoding data for a single display, FIG. 6 also indicates a seconddisplay input path SDVO2 216 that can perform similar processing stepsfor a second display input from a graphics and display controller 212,or from a second graphics and display controller (not shown). Anadvanced graphics and display controller 212 is designed with dual SDVOoutputs in order to support dual displays for a single user or tosupport very high resolution displays where a single SDVO port is notfast enough to handle the necessary data rate. The processing elementsof the multi-display processor 224 including the frame comparer 602, theframe manager 604, the data encoder 606 and the system controller 608can either be shared between the dual SDVO inputs, or a second set ofthe needed processing units can be included. If the processing isperformed by a programmable DSP or Media Processor, either a secondprocessor can be included or the one processor can be time multiplexedto manage both inputs.

The multi-display processor 224 outputs a display update stream to theFIG. 2 network controller 228 which in turn produces a display updatenetwork stream at one or more network interfaces 290. The networks maybe of similar or dissimilar nature but through the combination ofnetworks, each of the remote display systems, such as display system300, is accessible. High speed networks such as Gigabit Ethernet arepreferred but are not always practical. Lower speed networks such as10/100 Ethernet, Power Line Ethernet, Coaxial cable based Ethernet,phone line based Ethernet or wireless Ethernet standards such as802.11a, b, g, n, s and future derivatives can also be supported. Othernon-Ethernet connections are also possible and can include USB, 1394a,1394b, 1394c or other wireless protocols such as Ultra Wide Band (UWB)or WiMAX.

The various supported networks can support a variety of transmissionschemes. For example, Ethernet typically supports protocols such asstandard Transport Control Protocol and Internet Protocol (TCP/IP), UDP,some form of lightweight handshaking in combination with UDPtransmissions or a layer 2 based protocol. The performance of thenetwork connection will be one of the critical factors in determiningwhat resolution, color depth and frame rate can be supported for eachremote display system 300. Depending on the constraints of the networkenvironment, a proprietary protocol at an even lower level may be used.For example, a layer 2 protocol that does not use IP may perform anefficient transfer. Such a transfer may make use of the MAC addresses ofthe different devices. Special care needs to be taken with any routersand other network equipment to ensure such a scheme will work reliablyin the network topology being used.

The remote display performance can be optimized by matching the networkperformance and the display encoding dynamically in real time. Forexample, if the network congestion on one of the connections for one ofthe remote display systems increases at a point in time, themulti-display processor can be configured dynamically to reduce the datacreated for that remote display. When such a reduction becomesnecessary, the multi-display processor can reduce the display streamupdate data in various ways with the goal of having the least offensiveeffect on the quality of the display at the remote display system.Typically, the easiest adjustment is to lower the frame rate of displayupdates.

It is not typically possible or desirable to dynamically adjust theset-up of display resolution mode or display color depth mode of theremote display system as it would require a reconfiguration of thedisplay and the user would clearly find such an adjustmentobjectionable. However, depending on the data encoding method used, theeffective resolution and effective color depth within the existingdisplay format can be adjusted without the need to reconfigure thedisplay device and with a graceful degradation of the display quality.

Graceful degradation of this kind takes advantage of some of thecharacteristics of the human visual system's psycho-visual acuity where,when there are more changes and motion in the display, the psychovisional acuity is less sensitive to the sharpness of the picture. Forexample, when a person scrolls through a text document, his eye cannotfocus on the text as well as when the text is still, so that if the textblurred slightly during scrolling, it would not be particularlyobjectionable. Since the times of the most display stream updatescorrespond to the added motion on the display, it is at those times itmay be necessary to reduce the sharpness of the transmitted data inorder to lower the data rate. Such a dynamic reduction in sharpness canbe accomplished with a variety of encoding methods, but is particularlywell suited for Wavelet Transform based compression where the image issubband coded into different filtered and scaled versions of theoriginal image.

Multi-display processor 224 will detect when a frame input over the SDVObus intended for a remote display system is unchanged from the priorframe for that same remote display system. When the frame comparer 602detects such a sequence of unchanged frames, the data encoder 606 doesnot need to perform any encoding for that frame, the network controller228 will not generate a display update network stream for that frame,and the network bandwidth is conserved as the data necessary fordisplaying that frame already resides in the RAM 312 at the remotedisplay system 300. Similarly, no encoding is performed and no networktransmission is performed for identified precincts or groups of scanlines that the frame manager 604 and frame comparer 602 are able toidentify as unchanged. However, in each of these cases, the data wassent over the SDVO bus and may have been stored and read from RAM 230.

These SDVO transmissions and RAM movements would not be necessary if thehost computer 200 were able to track which display frames are beingupdated. Depending on the operating system it is possible for the CPU202 to track which frames for which displays are being updated. Thereare a variety of software based remote display Virtual Network Computing(VNC) products which use software running on the host system CPU 202 toreproduce the look of the display of a computer and can support viewingfrom a different type of platform and over low bandwidth connections.While conceptually interesting, this approach does not mimic a real timeresponse or support multi-media operations such as video and 3D that canbe supported by this preferred embodiment. However, a preferredembodiment of this invention can use software, combined with themulti-display processor hardware, to enhance the overall systemcapabilities.

VNC systems typically perform the operations of the graphics drawingengines using software on the host system CPU 202 because utilizing thegraphics and display controller 212 to perform the graphics drawingoperations would then require the CPU 202 to read back the resultingframes from the RAM 218 which would typically be inefficient. In onepreferred embodiment of this invention, the multi-display processor 224,through a combination of hardware and software, can perform theequivalent functions of the VNC encoding software and allow a VNCsoftware client to remotely view the accelerated frame buffer outputproduced by graphics and display controller 212. VNC typically supportsHextile encoded, ZRLE encoded and raw non-encoded formats and mayinclude 8, 64, 256 and full color options.

In some systems, the display encoding capabilities of the multi-displayprocessor 224 may be integrated into the network controller 228 or intoanother part of the host system. As such, the display data ports 214 and216 may not be available. In such a case, the main system bus 206 may beused to transfer the display frames or sub frames. The Graphics andDisplay Controller 212 would be configured to provide access to thedisplay frame data or would transfer the display frame data to anotherpart of the system over the main system bus 206. The display encodingfunction would then be performed on the display frame data by eitherdedicated hardware or a programmable processor. In one preferredembodiment of this invention, the network controller includes aprogrammable processor capable of performing the VNC Hextile, ZRLE andwavelet based encoding techniques. The Graphics and Display Controller212 provides the display frame data either over the main system bus 206or via a more dedicated bus 216. The processor within the networkcontroller (performs the display encoding and then via the networkcontrol functions provides the encoded display data over a network toeither a VNC client or to another remote display controller.

Various versions of Microsoft Windows® operating systems use GraphicsDevice Interface (GDI) calls for operations to the Graphics and DisplayController 212. Similarly, there are direct draw calls for controllingthe primary and secondary surface functions, direct 3D calls forcontrolling the 3D functions, and direct show calls for controlling thevideo playback related functions. Providing a tracking software layerthat either intercepts the various calls, or utilizing other utilitieswithin the display driver architecture, can enable the CPU subsystem 202to track which frames of which remote display system are being updated.By (performing this tracking, the CPU can reduce the need to sendunchanged frames over the SDVO bus. It would be further advantageous ifthe operating system or device driver support provided more directsupport for tracking which displays, which frames and which precinctswithin a frame had been modified. This operating system or device driverinformation could be used in a manner similar to the method describedfor the tracking software layer.

In a preferred embodiment, the CPU 202 can process data for more remotedisplay systems than the display control portion of the graphics anddisplay controller 212 is configured to support at any one time. Forexample, in the tiled display configuration for twelve simultaneousremote display systems of FIG. 4, additional displays could be swappedin and out of place of displays one through twelve based on the trackingsoftware layer. If the tracking software detected that no new activityhad occurred for display 5, and that a waiting list display 13 (notshown) had new activity, then CPU 202 would swap display 13 into theplace of display 5 in the tiled display memory area. CPU 202 may use the2D processor of the 2D, 3D and video graphics processors 410 to performthe swapping. A waiting list display 14 (not shown) could also replaceanother display such that the twelve shown displays are essentiallydisplay positions in and out of which the CPU 202 can swap an arbitrarynumber of displays. The twelve position illustration is arbitrary andthe system 100 could use as few as one or as many positions as themapping of the display sizes allows.

There are several considerations for using a tracking software layer forsuch a time multiplexing scheme. The display refresh operation ofdisplay controller-1 404 is asynchronous to the drawing by the 2D/3D andvideo processors 410 as well as asynchronous to the CPU 202 processes.This asynchronous nature makes it difficult for the multi-displayprocessor 224 to determine from the SDVO data if a display in the tileddisplay memory is the pre-swap display or the post-swap display. Worse,if the swap occurred during the read out of the tiled display regionbeing swapped, it would be possible for corrupted data to be output overSDVO. Synchronizing the swapping with the multi-display processor 224will require some form of semaphore operation, atomic operation,time-coordinated operation or software synchronization sequence.

The general software synchronization sequence is to inform themulti-display processor 224 that the display in (to use the example justabove) position 5 is about to b swapped and to not use the data fromthat position. The multi-display processor could still utilize data fromany of the other tiled display positions that were not being swapped.The CPU 202 and 2D/3D and video processor 410 will update the tileddisplay position with the new information for the swapped display. CPU202 then informs the multi-display processor 224 that data during thenext SDVO tiled display transfer would be from the new swapped displayand can be processed for the remote display system associated with thenew data. Numerous other methods of synchronization, including resettingthe display controller-1 404 to utilize another area of memory for thedisplay operations, are possible to achieve swapping benefits ofsupporting more users than there are hardware display channels at anyone time.

As described, it is possible to support more remote display systems 300than there are positions in the tiled display 406. The synchronizationoperations will take away some of the potential bandwidth for displayupdates, but overall, the system will be able to support more displays.In particular, one could consider a system 100 where many remotedisplays have little or no activity. In another system, where many ofthe remote displays do require frequent updates, the performance foreach remote display would be gracefully degraded through a combinationof reduced frame rate and reducing the visual detail of the contentwithin the display. If the system only included one display controller-1404, the group of six displays, 1 through 6, could be reconfigured suchthat the display controller would utilize the display memory associatedwith the group of six displays 7 through 12 for a time, then be switchedback.

The tiled method typically provides for the graphics and displaycontroller 212 providing the multi-display processor 224 with thecomplete frame information for each tile. There is also the ability toprovide sub-frame information via this tile approach provided that thesub-frame information relating to the position information of thesubframe is also provided. In a sub-framed method, instead of a completeframe occupying the tile, a number of subframes that can fit are fitinto the same area. Those sub-frames can all relate to one frame orrelate to multiple frames.

Another method to increase the number of remote displays supported is tobank switch the entire tile display area. For example, tilescorresponding to displays 1 through 6 may be refreshed over the SDVO1214 output while tiles corresponding to displays 7 through 12 are beingdrawn and updated. At the appropriate time, a bank switch occurs and thetiles for displays 7 through 12 become the active displays and tiles fordisplays 1 through 6 are then redrawn where needed. By switching all ofthe tiles in a bank at once, the number of synchronization steps may befewer than by switching each display independently.

To recap, by configuring and combining at a system level, the graphicsdisplay controller 212 with a multi-display processor 224 is able tosupport configurations varying in the number of remote display systems,resolution and color depth for each display, and the frame rateachievable by each display. An improved configuration could include fouror more SDVO output ports, and combined with the swapping procedure,could increase the ability of the system to support even more remotedisplay systems at higher resolutions. However, increasing the overallSDVO bandwidth and using dedicated memory and swapping for themulti-display processor comes at an expense in both increased systemcost and potentially increased system latency.

In an enhanced embodiment, not appropriate for all systems, it isdesirable to combine the multi-display processor with the system'sgraphics and video display controller and share the same memorysubsystem. FIG. 7 shows a preferred System-On-Chip (SOC) integratedcircuit embodiment 700 that combines the multi-user display capabilitiesalong with what is typically referred to as a graphics controller in acomputer system. The combined graphics controller with integratedmulti-display support 710 incorporates the functions of both graphicsand display controller 212 and multi-display processor 224 and as suchthis combined SOC would also connect to main system bus 206 in the hostcomputer 200 of a multi-display system 100 (FIG. 1).

In a preferred embodiment, the SOC graphics controller with integratedmulti-display support 710 includes a 2D Engine 720, 3D GraphicsProcessing Unit (GPU) 722, a system interface 732 such as PCI express,control for local I/O 728 that can include interfaces 730 for video orother local I/O, such as a direct interface to a network controller, anda memory interface 734. Additionally, the graphics controller mayinclude some combination of video compressor 724 and video decompression726 hardware, or some form of programmable video processor 764 thatcombines those and other video related functions.

While an embodiment can utilize the software driven GPU and videoprocessor approach for multi-display support as described above, theperformance of the system as measured by the frame rates for the numberof remote displays will be highest when using a graphics controller thatincludes a display subsystem optimized for multi-display processing.This further preferred embodiment 700 (FIG. 7) includes a multi-displayframe manager with display controller 750 and a display data encoder 752that compresses the display data. The multi-display frame manager withdisplay controller 750 may include outputs 756 and 758 for localdisplays, although the remote multi-display aspects are supported overthe system interface 732 or potentially a direct connection 730 to thenetwork controller. The system bus 760 is illustrative of theconnections between the various processing units as well as the systeminterface 732 and memory interface 734. The system bus 760 may includevarious forms of arbitrated transfers and may also have direct pathsfrom one unit to another for enhanced performance.

The multi-display frame manager with display controller 750 supportsfunctions similar to the FIG. 6 Frame Manager 604 and Frame Comparer 602of multi-display processor 224. By way of being integrated with thegraphics subsystem, some of the specific implementation capabilitiesimprove, although the previously described functions of managing themultiple display frames in memory, determining which frames have beenmodified by the CPU, running various graphics processors and videoprocessors, and managing the frames or blocks within the frames to beprocessed by the display data encoder 752 are generally supported.

In another embodiment of the present invention, the SOC graphicscontroller with integrated multi-display support 710 supports remoteclients that utilize a VNC type interface. To provide such support, thedisplay encoder 752 or another hardware block or CPU within integratedmulti-display support 710 performs some or all of the encoding typicallyperformed by the host CPU 210. An enhanced VNC client can utilize boththe traditional VNC commands as well as include decoding support forencoding enhancements that are supported within SOC 710. Such anapproach can allow both remote display systems 300 and remote PCs toperform remote computing tasks via the host system 200.

In the FIG. 2 multi-chip approach of host computer 200, the graphics anddisplay controller 212 is connected via the SDVO paths to themulti-display processor 224, and each controller and processor has itsown RAM system. Conversely, in FIG. 7 the graphics controller withintegrated multi-display support 710 uses the shared RAM 736 connectedby line 762 instead of the SDVO paths. Using the RAM 736 eliminates orreduces several bottlenecks in the system. First, the SDVO path transferbandwidth issue is eliminated. Second, by sharing the memory, themulti-display frame manager with display controller 750 is able to readthe frame information directly from the memory thus eliminating the readof memory by the graphics and display controller 212. For systems wherethe multi-display processor 224 was not performing operations on thefly, a write of the data into RAM is also eliminated.

Host computer 200 allows use of a graphics and display controller 212that may have not been designed for a multi-display system. Since thefunctional units within the graphics controller with integratedmulti-display support 710 may all be designed to be multi-display aware,additional optimizations can also be implemented. In a preferredembodiment, instead of implementing the multi-display frame support witha tiled display frame architecture, the multi-display frame manager withdisplay controller 750 may be designed to map support for multipledisplays that are matched as far as resolution and color depth in theircorresponding remote display systems.

By more directly matching the display in memory with the correspondingremote display systems, the swapping scheme described above can be muchmore efficiently implemented. Similarly, the tracking software layerdescribed earlier could be assisted with hardware that tracks when anypixels are changed in the display memory area corresponding to each ofthe displays. However, because a single display may include multiplesurfaces in different physical areas of memory, a memorycontroller-based hardware tracking scheme may not be the most economicalchoice.

In one embodiment of the present invention where hardware assistedtracking is utilized, the 2D engine 720, the 3D GPU 722 and the othervideo units that might change the display pixels utilize a memory basedbit scheme to set which pixels have been changed. The display encoder752 can be used to reset the tracking bit each time the pixels have beenencoded. This type of setting and resetting of a tracking bit can becombined with double buffering of surfaces. The tracking bits can beallocated for each pixel, or for a block of pixels. The bits can bestored in memory in proximity with the corresponding pixels or blocks ofpixels, or a separate memory surface may be used for the tracking bits.The hardware utilized by the 3D GPU 722 for setting the Z-buffering bitdepths or some other similarly designed hardware may be one way toimplement such a hardware assisted tracking mechanism.

The tracking software layer can also be used to assist in the encodingchoice for display frames that have changed and require generation of adisplay update stream. As discussed above the encoding is performed toreduce the data required for the remote display system 300 to regeneratethe display data generated by the host computer's graphics and displaycontroller 212. The tracking software layer can help identify the typeof data within a surface where the display controller-1 404 translatesthe surface into a portion of the display frame. That portion of thedisplay frame, whether precinct based or scan line based encoding isused, can be identified to the data encoder 606, or display data encoder752, as to allow the most optimal type of encoding to be performed.

For example, if the tracking software layer identifies that a surface isreal time video, then an encoding scheme more effective for video, whichhas smooth spatial transitions and temporal locality, can be used forthose areas of the frame. If the tracking software layer identifies thata surface is mostly text, then an encoding scheme more effective for thesharp edges and the ample white space of text can be used. Identifyingwhat type of data is in what region is a complicated problem. However,this embodiment of a tracking software layer allows an interface intothe graphics driver architecture of the host display system and hostoperating system that assists in this identification. For example, inMicrosoft Windows®, a surface that utilizes certain DirectShow commandsis likely to be video data whereas a surface that uses color expandingbit block transfers (Bit Blits) normally associated with text, is likelyto be text. Each operating system and graphics driver architecture willhave its own characteristic indicators. Other implementations canperform multiple types of data encoding in parallel and then, based onencoder feedback, choose to use the encoding scheme that produces thebest results.

Some types of encoding schemes are particularly more useful for specifictypes of data, and some encoding schemes are less susceptible to thetype of data. For example, RLE is very good for text and very poor forvideo, DCT based schemes are very good for video and very poor for text,and wavelet transform based schemes can do a good job for both video andtext. The VNC Hextile and ZRLE encoding techniques can be furthercombined with other encoding schemes for blocks that do not lendthemselves to Hextile and ZRLE. For example, if a block can not besufficiently encoded with Hextile, then that block may be encoded usinga Wavelet based scheme. Although any type of lossless or lossy encodingcan be used in this system, wavelet transform encoding, which also canbe of a lossless or lossy type, for this application has been describedin some detail. While optimizing the encoding based on the precinct isdesirable, it can not be used where it will cause visual artifacts atthe precinct boundaries or create other visual problems.

FIG. 8A illustrates the process of converting from a frame of videothrough to the communication processor. The first step is for eachcomponent of the video to be decomposed via subband encoding into amulti-resolution representation. The quad-tree-type decomposition forthe luminance component Y is shown in 812, for the first chrominancecomponent U in 814 and for the second chrominance component V in 816.The quad-tree-type decomposition splits each component into foursubbands where the first subband is represented by 818(h) 818(d) and818(v) with the h, d and v denoting horizontal, diagonal and vertical.The second subband, which is one half the first subband resolution inboth the horizontal and vertical direction, is represented in 820(h),820(d) and 820(v). The third subband is represented by 822(h), 822(d)and 822(v) and the fourth subband by box 824. Forward Error Correction(FEC) is an example of a method for improving the error resilience of atransmitted bitstream. FEC includes the process of adding additionalredundant bits of information to the base bits such that if some of thebits are lost or corrupted, the decoder system can reconstruct acomplete, or nearly complete, representation of the frame. The more bitsof redundant information that are added during the FEC step, the morestrongly protected, and the more resilient to errors the bit stream willbe. In the case of the wavelet encoded video, the lowest resolutionsubbands of the video frame may have the most image energy and can beprotected via more FEC redundancy bits than the higher resolutionsubbands of the frame.

In a typical network system, the FEC bits are used to protect a completepacket of information where each packet is protected by a checksum. Whenthe checksum properly arrives at the receiving end of a networktransmission, the packet of information can be assumed to be correct andthe packet is used. When the checksum arrives improperly, the packet isassumed to be corrupted and is not used. If packets of criticalinformation are corrupted, the network protocol may re-transmit them.For video, retransmission should be avoided as by the time aretransmitted packet is sent, it may be too late to be of use. Addingthe associated data traffic of retransmission can worsen a corruptedpacket situation. It is therefore desirable to assure that the moreimportant packets are more likely to arrive uncorrupted and that lessimportant packets, even if they are corrupted, are not retransmited. Theretransmission characteristics of a network can be managed in a varietyof ways including selection of TCP/IP and UDP style transmissions alongwith other network handshake operations.

The different subbands for each component are passed via path 802 to theencoding step. The encoding step is performed for each subband with theencoding with FEC performed on the first subband 836, on the secondsubband 834, on the third subband 832 and on the fourth subband 830.Depending on the type of encoding performed, there are various othersteps applied to the data prior to or as part of the encoding process.These steps can include filtering or differencing between the subbands.Encoding the differences between the subbands is one of the steps of atype of compression. For typical images, most of the image energyresides in the lower resolution representations of the image. The otherbands contain higher frequency detail that is used to enhance thequality of the image. The encoding steps for each of the subbands uses amethod and bitrate most suitable for the amount of visual detailcontained in that subimage.

There are also other scalable coding techniques that can used totransmit the different image subbands across different communicationchannels having different transmission characteristics. This techniquecan be used to match the higher priority source subbands with the higherquality transmission channels. This source based coding can be usedwhere the base video layer is transmitted in a heavily protected mannerand the upper layers are protected less or not at all. This can lead togood overall performance for error concealment and will allow forgraceful degradation of the image quality. Another technique of ErrorResilient Entropy Coding (EREC) can also be used for high resilience totransmission errors.

In addition to the dependence on the subimage visual detail, the type ofencoding and the strength of the FEC is dependent on the transmissionchannel error characteristics. The transmission channel feedback 840 isfed to the communications processor 844 which then feeds back theinformation via path 842 to each of the subband encoding blocks. Each ofthe subband encoders transmits the encoded subimage information to thecommunications processor 844. The communications processor 844 thentransmits the compressed streams via path 850 to the target transmissionsubsystem.

As an extension to the described 2-D subband coding, 3-D subband codingcan also be used. For 3-D subband coding, the subsampled component videosignals are decomposed into video components ranging from low spatialand temporal resolution components to components with higher frequencydetails. These components are encoded independently using the methodappropriate for preserving the image energy contained in the component.The compression is also performed independently through quantizing thevarious components and entropy coding of the quantized values. Thedecoding step is able to reconstruct the appropriate video image byrecovering and combining the various image components. A properlydesigned system, through the encoding and decoding of the video,preserves the psychovisual properties of the video image. Block matchingand block motion schemes can be used for motion tracking where the blocksizes may be smaller than the precinct size. Other advanced methods suchas applying more sophisticated motion coding techniques, imagesynthesis, or object-based coding are also possible.

Additional optimizations with respect to the transmission protocol arealso possible. For example, in one type of system there can be packetsthat are retransmitted if errors occur and there can be packets that arenot retransmitted regardless of errors. There are also various errorrate thresholds that can be set to determine if packets need to beresent. By managing the FEC allocation, along with the packettransmission protocol with respect to the different subbands of theframe, the transmission process can be optimized to assure that thedecoded video has the highest possible quality. Some types oftransmission protocols have additional channel coding that may bemanaged independently or combined with the encoding steps.

System level optimizations that specifically combine the subbandencoding with the UWB protocol are also possible. In one embodiment, thesubband with the most image energy utilizes the higher priority hardreservation scheme of the medium access control (MAC) protocol.Additionally, the low order band groups of the UWB spectrum thattypically have higher ranges can be used for the higher image energysubbands. In this case, even if a portable TV was out of range of theUWB high order band groups, the receiver would still receive the UWB loworder band groups and be able to display a moderate or low resolutionrepresentation of the original video.

FIG. 8B shows a tightly coupled two chip implementation of a Graphicsand Display Controller 212 utilizing an external Display EncoderIntegrated Circuit (IC) 803. The Display Encoder IC 803 is coupled tothe SDVO2 216 to receive uncompressed pixels into a pixel input port.The Display Encoder IC 803 then processes the pixels into an encodedform and outputs the encoded pixels on bus 804. Graphics and DisplayController 212 receives the encoded pixels over bus 804 to an inputport. The bus may be a high speed serial bus, a parallel bus or a busdesigned for another purpose that has adequate bandwidth to receive theencoded data. One example of such a bus is the CCIR656 bus which, whenoperated at an increased frequency, can be used for transferring theencoded data from the Display Encoder IC 803 to the Graphics and Displaycontroller 212.

As described earlier with respect to the Multi-Display Processor 224 andthe Display Encoder 752, the Display Encoder IC 803 may utilize avariety of internal buffering and external memory (not shown) schemes toperform a variety of encoding methods. The Graphics and DisplayController 212 may provide the non-encoded pixels over the SDVO2 216output in a variety of configurations including scan line or blockbased. The control signal and format information may be communicatedbetween the two chips 212 and 803 in a variety of ways including havingthe information embedded in the SDVO2 216 output stream, using a controlprotocol on either the SDVO2 line 216 or bus 804, or using a separatecommunication channel (not shown).

The Display Encoder IC 803 may have the full capabilities of theMulti-Display Processor 224 or of a subset of features of the FrameComparer 602, Frame Manager 604, Data Encoder 606, and System Controller608. Functions that are not included in the Display Encoder IC 803 maybe managed by a combination of the main system CPU 202 and the Graphicand Display Controller 212. For example, the SDVO2 output may be managedto only transmit frames or tiles that have already been determined to bedifferent from the prior frame or tile. As such, the Display Encoder IC803 would not need to perform the functions of the Frame Comparer 602 orthe Frame Manager 604. The Graphics and Display Controller 212, knowingwhich frames or tiles were transmitted to the Display Encoder IC 803,would know which encoded frames or tiles were being received.

The system 800 embodiment may be particularly well suited for a computerto support a second remote or wireless display system. In such anembodiment, the SDVO1 214 is connected to the internal display and theSDVO2 on line 216 output is used to support an external display. In oneconfiguration, the external display output SDVO2 216 is encoded by theDisplay Encoder IC 803, provided over bus 804 back to the Graphics andDisplay controller 212, transferred over the system bus 206 and outputover a network interface to a remote or wireless display system such asa large desktop LCD monitor that includes the necessary wireless anddecoder subsystems such as described in system 300.

FIG. 9A is a flowchart of method steps for performing one type ofcommunication between the host system and a Wireless Peripheral Device(WPD). At the start, the host system is already part of the WiFi networkand is in standard infrastructure mode. In this method, in step 902, theHost System and the WPD are set up and configured such thatcommunication between the two will be possible in the networkenvironment. The setup procedure may be automatic when the WPD powers onor may require putting the WPD into a special setup mode. The setup modemay be similar to the setup procedure performed by Bluetooth deviceswhere one typically presses a “connect device” button on the peripheraland runs a setup program on the host.

Unlike Bluetooth, which utilizes a different RF subsystem than thenetwork infrastructure, the host system communicates with the WiFiperipheral device utilizing the same RF subsystem and RF spectrum asused for communicating with the network infrastructure. As such, in step904, when the host initiates the setup procedure, it must firstsuccessfully arbitrate for the WiFi channel. Once the host has controlof the WiFi channel, in step 916 the host can perform whatever reads andwrites are required to perform the setup with the WPD and determine theprofile and capabilities of the WPD. The setup procedure may alsoinclude added security steps of entering a code or another security stepto assure that the WPD is the intended device. Part of the setup mayresult in the host system either learning the MAC address of the WPD orof the host system requesting an additional IP address from the networkinfrastructure, such as from the DHCP server, and relaying the assignedIP address to the WPD. In another setup procedure, once validated, theWPD is provided with the secure network key so that the WPD may accessthe secure network directly. Once the transfers between the host systemand the WPD have been completed, or once the RF channel needs to berelinquished, the flowchart follows path 920.

FIG. 9B illustrates the communication path 932 between the WiFi network930 and the Host System 200. Also shown is the communication path 936between the WPD 938 and the Host System 200. As illustrated by thisexample, the communication between the WPD 938 and the WIN Network 930takes place with the Host System 200 acting as the intermediary.Referring back to the flow chart of FIG. 9A, step 904 host arbitrationfor WiFi may be indirectly initiated by the WPD 938 through either apredetermined time interval or from another means. The WPD 938 operationof FIGS. 9A and 9B is for a WPD that does not include the capability todirectly interface to the WiFi network. Another type of WPD may alsoinclude a full or partial WiFi capability where the WPD may moredirectly participate in the WiFi network arbitration. Another type ofWPD may fully function on the WIN network when it is out of range of theHost System, but has another wired or wireless means to connect to theWiFi network.

FIG. 9C is a flowchart of method steps for performing the multi-displayprocessing procedure in accordance with one embodiment of the invention.For the sake of clarity, the procedure is discussed in reference todisplay data. However, procedures relating to audio and other data areequally contemplated for use in conjunction with the invention. In theFIG. 9C embodiment, initially, in step 910, multi-room television system200 and remote display system 300 follow the various procedures toinitialize and set up the host side and display side for the varioussubsystems to enable each display. In step 912, the graphics operationsof graphics display controller 212 process the various types of inputsto determine what graphics operations need to be performed. Thissimplified flow chart does not specifically call for the input from theremote display system 300 to be processed for determining the responsivegraphics operations, though another method would include those steps.

If graphics operations include 2D drawing, then, in step 924, the 2Ddrawing engine 720 or associated function unit of graphics and displayprocessor 212 preferably processes the operations into the appropriatedisplay surface in the appropriate RAM. Similarly, in step 926 3Ddrawing is performed to the appropriate display surface in RAM by eitherthe 3D GPU 722 or the associated unit in graphics and display processor212. Similarly, in step 928, video rendering is performed to theappropriate display surface in RAM by one of the video processing units724, 726 or the associated units in graphics and display processor 212.Though not shown, any CPU subsystem 202-initiated drawing operations tothe RAM would occur at this stage of the flow as well.

The system in step 940 composites the multiple surfaces into a singleimage frame which is suitable for display. This compositing can beperformed with any combination of operations by the CPU subsystem 202,2D engine 720, 3D GPU 722, video processing elements 724, 726 or 764,multi-display frame manager with display controller 750 or thecomparable function blocks of graphics and display controller 212. Oncethe compositing operation is performed, step 946 performs the framemanagement with the frame manager 604 or multi-display frame managerwith display controller 750 which includes tracking the frame updatesfor each remote display. Then step 950 compares the frame to theprevious frame for that same remote display system via a combination ofthe software tracking layer, combined with frame comparer 602 or themulti-display frame manager with display controller 750. The compareframe step 950 identities which areas of each frame need to be updatedfor the remote displays where the areas can be identified by precincts,scan line groups or another manner.

The system, in step 954, then encodes the data that requires the updatevia a combination of software and data encoder 606 or display dataencoder 752. The data encoding step 954 can use the tracking software toidentify what type of data is going to be encoded so that the mostefficient method of encoding is selected or the encoding hardware canadaptively perform the encoding without any knowledge of the data.Feedback path 968 from the network process step 962 may be used by theencode data step 954 in order to more efficiently encode the data todynamically match the encoding to the characteristics of the networkchannel. This may include adjustments to the compression ratio as wellas to the error resilience of the encoded data. The error resilience mayidentify different priorities of data within the encoded data stream forany given display which is used by in step 958 to perform the systemcontrol for each frame update.

The system, in step 958, utilizes the encoded data information and theassociated system information to manage the frame updates to the remotedisplays. The system control step 958 also utilizes the networktransmission channel information via feedback path 968 to manage andselect some of the higher level network decisions. This system controlstep is performed with some combination of the CPU subsystem 202 andsystem controller unit 608 or multi-display frame manager with displaycontroller 750.

The network process step 962 uses the information from the systemcontrol. This information can include information as to which remotedisplay requires which frame update streams, what type of networktransmission protocol is used for each frame update stream, and what thepriority and retry characteristics are for each portion of each frameupdate stream. The network process step 962 utilizes the networkcontroller 228 to manage any number of network connections. The variousnetworks may include Gigabit Ethernet, 10/100 Ethernet, Power LineEthernet, Coaxial cable based Ethernet, phone line based Ethernet, orwireless Ethernet standards such as 802.11a, b, g, n, s and futurederivatives. Other non-Ethernet connections are also possible and caninclude USB, 1394a, 1394b, 1394c or other wireless protocols such asUltra Wide Band (UWB) or WiMAX.

FIG. 10 is a flowchart of steps in a method for performing a networkreception and display procedure in accordance with one embodiment of theinvention. For reasons of clarity, the procedure is discussed inreference to display data. However, procedures relating to audio andother data are equally contemplated for use in conjunction with thepresent invention.

In the FIG. 10 embodiment, initially, in step 1012, remote displaysystem 300 preferably receives a frame update stream from host computer200 of a multi-display system 100. Then, in step 1014, networkcontroller 326 preferably performs a network processing procedure toexecute the network protocols to receive the transmitted data whetherthe transmission was wired or wireless.

In step 1020, data decoder and frame manager 328 receives and preferablymanipulates the data information into an appropriate displayable format.In step 1030, data decoder and frame manager 328 preferably may accessthe data manipulated in step 1020 and produce an updated display frameinto RAM 312. The updated display frame may include display frame datafrom prior frames, the manipulated and decoded new frame data, and anyprocessing required to conceal display data errors that occurred duringtransmission of the new frame data.

Finally, in step 1040, display controller 330 provides the most recentlycompleted display frame data to remote display screen 310 for viewing bya user of the remote display system 300. In the absence of either ascreen saving or power down mode, the display processor will continue toupdate the remote display screen 310 with the most recently completeddisplay frame, as indicated with feedback path 1050, in the process ofdisplay refresh.

The invention has been explained above with reference to a preferredembodiment. Other embodiments will be apparent to those skilled in theart in light of this disclosure. For example, the present invention mayreadily be implemented using configurations other than those describedin the preferred embodiment above. Additionally, the present inventionmay effectively be used in conjunction with systems other than the onedescribed above as the preferred embodiment. Therefore, these and othervariations upon the preferred embodiments are intended to be covered bythe present invention, which is limited only by the appended claims.

I claim:
 1. A method comprising: establishing a first Wi-Fi connectionbetween a host computing device and a peripheral device, wherein thefirst Wi-Fi connection provides a direct wireless communication linkbetween the host computing device and the peripheral device via a Wi-Fiperipheral mode that differs from a Wi-Fi infrastructure mode and aWi-Fi ad hoc mode; establishing, via the Wi-Fi infrastructure mode, asecond Wi-Fi connection between the host computing device and a Wi-Fiaccess point that is unaware of the Wi-Fi peripheral mode; andtransmitting, via the direct wireless communication link of the firstWi-Fi connection, content between the host computing device and theperipheral device per the Wi-Fi peripheral mode such that the Wi-Fiaccess point views the host computing device as operating per the Wi-Fiinfrastructure mode.
 2. The method of claim 1, wherein said establishingthe first Wi-Fi connection between the host computing device and theperipheral device comprises exchanging a key between the host computingdevice and the peripheral device.
 3. The method of claim 2, wherein saidexchanging the key comprises exchanging the key via the first Wi-Ficonnection between the host computing device and the peripheral device.4. The method of claim 2, wherein said exchanging the key comprisesexchanging the key between the host computing device and the Wi-Fiaccess point via the second Wi-Fi connection.
 5. The method of claim 2,wherein said establishing the first Wi-Fi connection between the hostcomputing device and the peripheral device comprises establishing theWi-Fi connection between the host computing device and the peripheraldevice without receiving a key entered by a user.
 6. The method of claim2, wherein said exchanging the key between the host computing device andthe peripheral device comprises receiving, at the peripheral device, thekey from the sccond host computing device.
 7. The method of claim 1,wherein said establishing the first Wi-Fi connection between the hostcomputing device and the peripheral device comprises establishingwireless communication between a mobile device and a television.
 8. Themethod of claim 1, wherein the host computing device and is capable ofconcurrently wirelessly communicating over the first Wi-Fi connectionand the second Wi-Fi connection.
 9. The method of claim 1 furthercomprising concurrently wirelessly coupling the peripheral device, viathe host computing device, to the second Wi-Fi connection and the Wi-Fiaccess point.
 10. The method of claim 1 further comprising wirelesslycoupling the host computing device and the peripheral device to theWi-Fi access point while maintaining the first Wi-Fi connection betweenthe host computing device and the peripheral device.
 11. The method ofclaim 1, wherein the host computing device streams to the peripheraldevice display content from a third device.
 12. A method comprising:establishing a wireless peripheral mode connection between a networkcontroller of a host system and a wireless peripheral device, whereinthe wireless peripheral mode connection provides a direct wirelesscommunication link between the host system and the wireless peripheraldevice, and wherein the wireless peripheral mode connection utilizes asingle RF subsystem of the network controller and an RF channel of thesingle RF subsystem; establishing a wireless infrastructure modeconnection between the network controller of the host system and awireless access point of a wireless network, wherein the wirelessinfrastructure mode connection utilizes the single RF subsystem of thenetwork controller and the RF channel of the single RF subsystem; andtransmitting content, via the direct wireless communication link and aMAC address based protocol, between the wireless peripheral device andthe host system via the established wireless peripheral mode connectionin a manner that is viewed by the wireless access point as aninfrastructure mode transmission that is effectively ignored by wirelessaccess point.
 13. The method of claim 12 further comprising: validatingthe wireless peripheral device to the host system via the establishedwireless peripheral mode connection, wherein said validating assures anidentity of the wireless peripheral device to the host system; and aftersaid validating, transferring, from the host system to the wirelessperipheral device, a network key used to secure the wireless network.14. The method of claim 12, further comprising: placing the wirelessperipheral device in a sleep mode; and managing, with the host system,an infrastructure mode wireless arbitration protocol to obtain controlof the RF channel, used by both the wireless peripheral mode connectionand the wireless infrastructure mode connection, on behalf of thewireless peripheral device while the wireless peripheral device is inthe sleep mode, wherein said transmitting content includes transmittingthe content between the host system and the wireless peripheral deviceafter the host system obtains control of the RF channel on behalf of thewireless peripheral device.
 15. The method of claim 12, furthercomprising: managing, with the host system, an infrastructure modearbitration protocol to obtain control of the RF channel, used by boththe wireless peripheral mode connection and the wireless infrastructuremode connection, on behalf of the wireless peripheral device; and wakingthe wireless peripheral device after host system obtains control of theRF channel, wherein said transmitting comprises transmitting the contentbetween the host system and the wireless peripheral device after saidwaking.
 16. The method of claim 12, further comprising displaying thecontent via a display of the wireless peripheral device.
 17. The methodof claim 12, further comprising wirelessly streaming additional contentfrom the wireless network to the wireless peripheral device.
 18. Themethod of claim 12, further comprising wirelessly streaming additionalcontent from the wireless network to the wireless peripheral device viathe host system, the wireless infrastructure mode connection, and thewireless peripheral mode connection.
 19. The method of claim 12, whereinthe wireless peripheral device is a keyboard.
 20. The method of claim12, wherein the wireless peripheral device is a mouse.